{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b44c6176-a0ce-43cb-8f0a-20eeaa43fe35",
   "metadata": {},
   "outputs": [],
   "source": [
    "数值的稳定性与模型的初始化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0f8ad2c-664d-49d9-819f-5e8876c61b72",
   "metadata": {},
   "outputs": [],
   "source": [
    "深度模型有关数值稳定性的典型问题是衰减和爆炸\n",
    "当出现很多层神经网络时，例如权重参数为超参数0.2与5时，当神经网络有20层，则参数变为\n",
    "0.2的20次方，称为衰减，5称为爆炸"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edaab40d-e85a-4cfc-a6d0-1d7c18bfa6fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "随机初始化模型参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0c98e79-60da-419f-aac6-561fa41b3ecc",
   "metadata": {},
   "outputs": [],
   "source": [
    "如果隐藏层使用相同的激活函数，将每个隐藏单元的参数都初始化为相等的值，正向传播中得到的输入计算值相同\n",
    "反向传播中，每个隐藏单元的参数梯度也相同。在使用梯度优化算法迭代后的值仍然相同。因此，需要对模型进行随机初始化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a15546d6-613b-4df8-ba64-169e43e313f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "Xavier随机初始化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9f292f8-74d2-4268-96aa-01abeec78e99",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:pytorch]",
   "language": "python",
   "name": "deep_learning"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
